{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "$$Hyperopt tutorials： Xgboost + CV$$\n",
    "\n",
    "## 前言\n",
    "Xgboost中内置了交叉验证，如果我们需要使用交叉验证的话，只需要直接调用即可。我们依旧采用第一篇教程使用过代码。你可以直接跳到交叉验证部分。\n",
    "\n",
    "与[第一篇教程-如何使用hyperopt对xgboost进行自动调参](https://blog.csdn.net/FontThrone/article/details/85045254)相同的是，本处教程中的代码也可以很好的被套用。并且可以很好的迁移到lightgbm与catboost上面。\n",
    "\n",
    "## 加载数据\n",
    "### 读取数据并分割\n",
    "\n",
    "因为我们这里使用的是交叉验证因此我们也就不再需要，将数据集分割为三份了，只需要分割出百分之十的数据用于预测就好。注意随机数的问题。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:24:30.987282Z",
     "start_time": "2019-04-12T10:24:30.216301Z"
    }
   },
   "outputs": [],
   "source": [
    "from hyperopt import fmin, tpe, hp, partial\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split, cross_val_score\n",
    "from sklearn.metrics import mean_squared_error, zero_one_loss\n",
    "import xgboost as xgb\n",
    "import pandas as pd\n",
    "\n",
    "def GetNewDataByPandas():\n",
    "    wine = pd.read_csv(\"../data/wine.csv\")\n",
    "    wine['alcohol**2'] = pow(wine[\"alcohol\"], 2)\n",
    "    wine['volatileAcidity*alcohol'] = wine[\"alcohol\"] * wine['volatile acidity']\n",
    "    y = np.array(wine.quality)\n",
    "    X = np.array(wine.drop(\"quality\", axis=1))\n",
    "\n",
    "    columns = np.array(wine.columns)\n",
    "\n",
    "    return X, y, columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 分割数据并转换\n",
    "首先将数据分割为三份，一部分用于预测，训练数据则同样分成额外的两部分用于evallist参数。\n",
    "\n",
    "同时为了加快速度和减少内存，我们将数据转换为`xgboost`自带的读取格式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:24:32.047453Z",
     "start_time": "2019-04-12T10:24:32.003546Z"
    }
   },
   "outputs": [],
   "source": [
    "# Read wine quality data from file\n",
    "X, y, wineNames = GetNewDataByPandas()\n",
    "\n",
    "# split data to [[0.8,0.2],01]\n",
    "x_train_all, x_predict, y_train_all, y_predict = train_test_split(X, y, test_size=0.10, random_state=100)\n",
    "\n",
    "x_train, x_test, y_train, y_test = train_test_split(x_train_all, y_train_all, test_size=0.2, random_state=100)\n",
    "\n",
    "dtrain = xgb.DMatrix(data=x_train,label=y_train,missing=-999.0)\n",
    "dtest = xgb.DMatrix(data=x_test,label=y_test,missing=-999.0)\n",
    "\n",
    "evallist = [(dtest, 'eval'), (dtrain, 'train')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 利用交叉验证训练\n",
    "### 训练模型\n",
    "\n",
    "在之前的代码中，我将数据分割为 6:3:1，其分别为，训练数据，性能监视用数据，和最后的预测数据。这个比例只是为了示例用，并不具有代表性。\n",
    "\n",
    "本处则主要介绍交叉验证方法。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 交叉验证与Hyperopt结合\n",
    "\n",
    "    xgboost进行交叉验证与Hyperopt结合有两种方案，第一种方案是使用本身自带的CV方法，但是这种方案的存在一个问题，就是CV中无法直接传递分开的参数，而只能传递唯一参数params，因此我们需要先生成一个model，然后通过`get_params()`来获取参数，这种方式代码会稍微多几行。不过相较于与sklearn结合的形式，计算时间上则有很大的提升。\n",
    "    \n",
    "    \n",
    "    第二种方案：也可以直接使用`sklearn.model_selection`中的多种交叉验证方案，只将xgboost作为单个模型传入。但是个人并不建议这样做。首先是和sklearn的交互会一定程度上导致计算性能的下降，而且计算时间上差距可能会很大。所以个人建议只有在必要的时候，可以采用最传统的`train_test_split`方案时使用类似的代码结构，但是还是不适用和sklearn结合形式的交叉验证，具体怎么做会在其他例子中介绍。\n",
    "    \n",
    "    \n",
    "### 使用CV方法进行交叉验证\n",
    "\n",
    "    直接使用xgboost中的cv方法进行交叉验证应该是最好的方案。唯一的问题是如何传入唯一的`params`参数。不知道为什么在我翻译玩hyperopt中文文档的这近一年，都没有见到有人写这一类的。有可能是没想到如何解决吧这个问题吧。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:26:33.771823Z",
     "start_time": "2019-04-12T10:26:33.766865Z"
    }
   },
   "outputs": [],
   "source": [
    "import hyperopt\n",
    "\n",
    "def hyperopt_objective(params):\n",
    "    \n",
    "    model = xgb.XGBRegressor(\n",
    "        max_depth=int(params['max_depth'])+5,\n",
    "        learning_rate=params['learning_rate'],\n",
    "        silent=1,\n",
    "        objective='reg:linear',\n",
    "        eval_metric='rmse',\n",
    "        seed=42,\n",
    "        nthread=-1,\n",
    "    )\n",
    "     \n",
    "    res = xgb.cv(model.get_params(), dtrain, num_boost_round=10, nfold=5,\n",
    "             callbacks=[xgb.callback.print_evaluation(show_stdv=False),\n",
    "                        xgb.callback.early_stop(3)])\n",
    "    \n",
    "    return np.min(res['test-rmse-mean']) # as hyperopt minimises"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:26:43.526731Z",
     "start_time": "2019-04-12T10:26:34.179758Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0]\ttrain-rmse:3.24774\ttest-rmse:3.24823                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.07359\ttest-rmse:2.0891                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.36012\ttest-rmse:1.40987                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.928331\ttest-rmse:1.02378                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.657333\ttest-rmse:0.827554                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.488443\ttest-rmse:0.732572                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.382361\ttest-rmse:0.690592                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.310456\ttest-rmse:0.669706                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.257609\ttest-rmse:0.661822                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.216769\ttest-rmse:0.657073                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.09603\ttest-rmse:3.09669                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.89483\ttest-rmse:1.91313                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.21091\ttest-rmse:1.25043                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.836588\ttest-rmse:0.908363                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.640539\ttest-rmse:0.755092                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.539681\ttest-rmse:0.688215                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.48786\ttest-rmse:0.657656                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.457719\ttest-rmse:0.649595                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.438881\ttest-rmse:0.646575                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.42562\ttest-rmse:0.644122                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:3.24957\ttest-rmse:3.25005                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.0758\ttest-rmse:2.09129                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.36267\ttest-rmse:1.41221                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.936506\ttest-rmse:1.02038                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.679966\ttest-rmse:0.820449                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.529603\ttest-rmse:0.728334                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.436567\ttest-rmse:0.680488                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.380164\ttest-rmse:0.660876                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.34202\ttest-rmse:0.654416                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.318009\ttest-rmse:0.649212                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.40573\ttest-rmse:3.40607                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.27081\ttest-rmse:2.28065                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.54574\ttest-rmse:1.57119                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.09153\ttest-rmse:1.14206                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.811721\ttest-rmse:0.894493                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.647758\ttest-rmse:0.765045                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.552754\ttest-rmse:0.700553                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.49612\ttest-rmse:0.668229                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.46416\ttest-rmse:0.653895                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.44308\ttest-rmse:0.647017                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:4.27349\ttest-rmse:4.27337                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.5374\ttest-rmse:3.5404                                                                                 \n",
      "\n",
      "[2]\ttrain-rmse:2.93451\ttest-rmse:2.93692                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.44122\ttest-rmse:2.44883                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:2.0379\ttest-rmse:2.05151                                                                                \n",
      "\n",
      "[5]\ttrain-rmse:1.70858\ttest-rmse:1.73018                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.43989\ttest-rmse:1.47389                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:1.22145\ttest-rmse:1.26952                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:1.0439\ttest-rmse:1.10779                                                                                \n",
      "\n",
      "[9]\ttrain-rmse:0.898214\ttest-rmse:0.986378                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.24952\ttest-rmse:4.2494                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.49834\ttest-rmse:3.50163                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.88671\ttest-rmse:2.88959                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.38934\ttest-rmse:2.39843                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.98544\ttest-rmse:2.00213                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.65759\ttest-rmse:1.67981                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.39265\ttest-rmse:1.42613                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:1.17851\ttest-rmse:1.22836                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:1.00569\ttest-rmse:1.07321                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:0.867835\ttest-rmse:0.954791                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.76117\ttest-rmse:3.76126                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.75186\ttest-rmse:2.75808                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.03055\ttest-rmse:2.04168                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.51784\ttest-rmse:1.55028                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.15496\ttest-rmse:1.20905                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.896884\ttest-rmse:0.987911                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.709957\ttest-rmse:0.849125                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.576018\ttest-rmse:0.758931                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.478092\ttest-rmse:0.706351                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.411829\ttest-rmse:0.677039                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.10422\ttest-rmse:3.10487                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.90395\ttest-rmse:1.9237                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.2121\ttest-rmse:1.27141                                                                                \n",
      "\n",
      "[3]\ttrain-rmse:0.810284\ttest-rmse:0.935069                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.567049\ttest-rmse:0.779285                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.411852\ttest-rmse:0.703612                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.315433\ttest-rmse:0.673708                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.24909\ttest-rmse:0.66296                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:0.206719\ttest-rmse:0.655243                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.171989\ttest-rmse:0.653996                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.95679\ttest-rmse:4.95657                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:4.75173\ttest-rmse:4.75161                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:4.55544\ttest-rmse:4.55547                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:4.3676\ttest-rmse:4.36794                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:4.18777\ttest-rmse:4.18808                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:4.01574\ttest-rmse:4.01601                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:3.85106\ttest-rmse:3.85182                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:3.69351\ttest-rmse:3.69494                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:3.54274\ttest-rmse:3.54407                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:3.39848\ttest-rmse:3.40055                                                                               \n",
      "\n",
      "[0]\ttrain-rmse:3.60052\ttest-rmse:3.60071                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.52779\ttest-rmse:2.53699                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.79732\ttest-rmse:1.81791                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.30161\ttest-rmse:1.34956                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.96503\ttest-rmse:1.05219                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.73053\ttest-rmse:0.871645                                                                              \n",
      "\n",
      "[6]\ttrain-rmse:0.570272\ttest-rmse:0.767151                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.454064\ttest-rmse:0.712563                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.374496\ttest-rmse:0.677264                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.317389\ttest-rmse:0.660278                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.63967\ttest-rmse:4.63948                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:4.16484\ttest-rmse:4.16576                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:3.74074\ttest-rmse:3.74172                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:3.36187\ttest-rmse:3.3639                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:3.0238\ttest-rmse:3.02658                                                                                \n",
      "\n",
      "[5]\ttrain-rmse:2.72199\ttest-rmse:2.72804                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:2.45274\ttest-rmse:2.4604                                                                                \n",
      "\n",
      "[7]\ttrain-rmse:2.21255\ttest-rmse:2.2235                                                                                \n",
      "\n",
      "[8]\ttrain-rmse:1.99865\ttest-rmse:2.01448                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:1.80801\ttest-rmse:1.82712                                                                               \n",
      "\n",
      "[0]\ttrain-rmse:4.16834\ttest-rmse:4.16825                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.3675\ttest-rmse:3.37087                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:2.72888\ttest-rmse:2.73405                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.22026\ttest-rmse:2.2316                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:1.81627\ttest-rmse:1.83603                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.49608\ttest-rmse:1.52433                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.24428\ttest-rmse:1.28482                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:1.04733\ttest-rmse:1.10197                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:0.894097\ttest-rmse:0.964754                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.775672\ttest-rmse:0.864907                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.22373\ttest-rmse:3.22425                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.04455\ttest-rmse:2.06079                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.33473\ttest-rmse:1.38606                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.915019\ttest-rmse:0.999943                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.665926\ttest-rmse:0.805452                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.519146\ttest-rmse:0.714662                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.428857\ttest-rmse:0.674705                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.379837\ttest-rmse:0.657131                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.337581\ttest-rmse:0.650166                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.312429\ttest-rmse:0.644884                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.63665\ttest-rmse:3.63682                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.57723\ttest-rmse:2.5862                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.84779\ttest-rmse:1.86647                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.34685\ttest-rmse:1.39049                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.00189\ttest-rmse:1.08384                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.759899\ttest-rmse:0.898483                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.588727\ttest-rmse:0.784045                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.46077\ttest-rmse:0.717876                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.368999\ttest-rmse:0.680802                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.300124\ttest-rmse:0.659593                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.56591\ttest-rmse:3.56612                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.48095\ttest-rmse:2.49045                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.75042\ttest-rmse:1.77137                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.26306\ttest-rmse:1.30498                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.941575\ttest-rmse:1.01502                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.728484\ttest-rmse:0.840947                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.594862\ttest-rmse:0.743458                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.505992\ttest-rmse:0.688756                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.448073\ttest-rmse:0.662649                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.413055\ttest-rmse:0.644251                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.88577\ttest-rmse:2.88671                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.66406\ttest-rmse:1.68422                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.02893\ttest-rmse:1.07531                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.715118\ttest-rmse:0.80584                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.56666\ttest-rmse:0.705066                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.499136\ttest-rmse:0.670988                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.460227\ttest-rmse:0.660647                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.436291\ttest-rmse:0.660154                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.41635\ttest-rmse:0.659003                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.403537\ttest-rmse:0.657696                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.71889\ttest-rmse:4.71869                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:4.30772\ttest-rmse:4.30791                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:3.93374\ttest-rmse:3.93489                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:3.5937\ttest-rmse:3.59499                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:3.28484\ttest-rmse:3.28729                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:3.0041\ttest-rmse:3.00863                                                                                \n",
      "\n",
      "[6]\ttrain-rmse:2.74879\ttest-rmse:2.7556                                                                                \n",
      "\n",
      "[7]\ttrain-rmse:2.51695\ttest-rmse:2.52572                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:2.30631\ttest-rmse:2.31993                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:2.11517\ttest-rmse:2.13088                                                                               \n",
      "\n",
      "[0]\ttrain-rmse:3.58813\ttest-rmse:3.58833                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.51097\ttest-rmse:2.52027                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.78036\ttest-rmse:1.80296                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.28864\ttest-rmse:1.33139                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.959717\ttest-rmse:1.03767                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.739029\ttest-rmse:0.859226                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.59446\ttest-rmse:0.754674                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.495451\ttest-rmse:0.69746                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.428136\ttest-rmse:0.666114                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.382377\ttest-rmse:0.650354                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.09567\ttest-rmse:4.0956                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.25258\ttest-rmse:3.25564                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.59257\ttest-rmse:2.5969                                                                                \n",
      "\n",
      "[3]\ttrain-rmse:2.07729\ttest-rmse:2.08858                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.67562\ttest-rmse:1.69671                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.36412\ttest-rmse:1.39713                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.12576\ttest-rmse:1.17366                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:0.941447\ttest-rmse:1.00536                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.80386\ttest-rmse:0.884966                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.699638\ttest-rmse:0.79937                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:4.09728\ttest-rmse:4.09721                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.2551\ttest-rmse:3.25815                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:2.59537\ttest-rmse:2.59938                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.07977\ttest-rmse:2.09217                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.67726\ttest-rmse:1.70042                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.36373\ttest-rmse:1.3982                                                                                \n",
      "\n",
      "[6]\ttrain-rmse:1.1198\ttest-rmse:1.17241                                                                                \n",
      "\n",
      "[7]\ttrain-rmse:0.929419\ttest-rmse:1.00778                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.783317\ttest-rmse:0.889186                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.668064\ttest-rmse:0.803436                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.684\ttest-rmse:2.68527                                                                                 \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.45991\ttest-rmse:1.48818                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:0.875391\ttest-rmse:0.944776                                                                             \n",
      "\n",
      "[3]\ttrain-rmse:0.606956\ttest-rmse:0.741324                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.494224\ttest-rmse:0.674932                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.437749\ttest-rmse:0.653196                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.408117\ttest-rmse:0.648024                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.386697\ttest-rmse:0.647981                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.360255\ttest-rmse:0.647513                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.34679\ttest-rmse:0.646486                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:2.66562\ttest-rmse:2.66693                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.44237\ttest-rmse:1.47108                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:0.863307\ttest-rmse:0.934505                                                                             \n",
      "\n",
      "[3]\ttrain-rmse:0.599674\ttest-rmse:0.736169                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.492081\ttest-rmse:0.676391                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.431637\ttest-rmse:0.660436                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.399188\ttest-rmse:0.6551                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:0.376036\ttest-rmse:0.655284                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.356145\ttest-rmse:0.657623                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.337836\ttest-rmse:0.65744                                                                              \n",
      "\n",
      "Stopping. Best iteration:                                                                                              \n",
      "[6]\ttrain-rmse:0.399188+0.0157577\ttest-rmse:0.6551+0.0503749\n",
      "\n",
      "\n",
      "[0]\ttrain-rmse:2.92221\ttest-rmse:2.9231                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1]\ttrain-rmse:1.70245\ttest-rmse:1.7207                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.05225\ttest-rmse:1.10725                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.719103\ttest-rmse:0.827471                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.560059\ttest-rmse:0.717657                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.471291\ttest-rmse:0.676371                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.422191\ttest-rmse:0.658176                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.395564\ttest-rmse:0.649618                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.375329\ttest-rmse:0.650981                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.350742\ttest-rmse:0.653761                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.01569\ttest-rmse:3.01645                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.80413\ttest-rmse:1.82267                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.13298\ttest-rmse:1.18544                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.772385\ttest-rmse:0.872282                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.582095\ttest-rmse:0.737447                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.481356\ttest-rmse:0.686626                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.423933\ttest-rmse:0.667904                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.388681\ttest-rmse:0.662                                                                                \n",
      "\n",
      "[8]\ttrain-rmse:0.366626\ttest-rmse:0.659392                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.34478\ttest-rmse:0.655514                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:3.40329\ttest-rmse:3.40363                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.2677\ttest-rmse:2.27756                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.54281\ttest-rmse:1.56832                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.08918\ttest-rmse:1.14044                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.81098\ttest-rmse:0.892452                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.647934\ttest-rmse:0.762117                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.552796\ttest-rmse:0.699668                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.496453\ttest-rmse:0.667426                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.4642\ttest-rmse:0.653022                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:0.445459\ttest-rmse:0.647112                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.88546\ttest-rmse:3.88548                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.93243\ttest-rmse:2.93648                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.22708\ttest-rmse:2.2345                                                                                \n",
      "\n",
      "[3]\ttrain-rmse:1.70773\ttest-rmse:1.73085                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.32658\ttest-rmse:1.36617                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.05012\ttest-rmse:1.11016                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:0.849754\ttest-rmse:0.937246                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.704974\ttest-rmse:0.821228                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.603432\ttest-rmse:0.751165                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.533159\ttest-rmse:0.704925                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.79461\ttest-rmse:2.79569                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.56865\ttest-rmse:1.59222                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:0.943264\ttest-rmse:1.02038                                                                              \n",
      "\n",
      "[3]\ttrain-rmse:0.611229\ttest-rmse:0.792978                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.434693\ttest-rmse:0.715105                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.332221\ttest-rmse:0.685095                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.27023\ttest-rmse:0.675704                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.230749\ttest-rmse:0.670791                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.199329\ttest-rmse:0.669553                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.170616\ttest-rmse:0.670054                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.45281\ttest-rmse:3.45312                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.33146\ttest-rmse:2.34082                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.6019\ttest-rmse:1.63276                                                                                \n",
      "\n",
      "[3]\ttrain-rmse:1.13098\ttest-rmse:1.1919                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:0.825339\ttest-rmse:0.93396                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.628375\ttest-rmse:0.788776                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.49991\ttest-rmse:0.714092                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.412312\ttest-rmse:0.681816                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.351005\ttest-rmse:0.665549                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.300936\ttest-rmse:0.656367                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.2232\ttest-rmse:3.22372                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.04391\ttest-rmse:2.06016                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.33359\ttest-rmse:1.38626                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.906827\ttest-rmse:1.00412                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.640704\ttest-rmse:0.817193                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.473847\ttest-rmse:0.727776                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.373325\ttest-rmse:0.685411                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.300906\ttest-rmse:0.669272                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.257055\ttest-rmse:0.662622                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.222631\ttest-rmse:0.659612                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.95941\ttest-rmse:3.9594                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.04299\ttest-rmse:3.04686                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.35106\ttest-rmse:2.35728                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.83094\ttest-rmse:1.84677                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.4418\ttest-rmse:1.47076                                                                                \n",
      "\n",
      "[5]\ttrain-rmse:1.15246\ttest-rmse:1.19952                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:0.939954\ttest-rmse:1.00532                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.785723\ttest-rmse:0.872138                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.674651\ttest-rmse:0.781921                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.596792\ttest-rmse:0.725188                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.10212\ttest-rmse:3.10277                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.9016\ttest-rmse:1.92141                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.21393\ttest-rmse:1.26018                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.830257\ttest-rmse:0.917176                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.620341\ttest-rmse:0.758851                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.509671\ttest-rmse:0.692851                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.44865\ttest-rmse:0.66696                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:0.407247\ttest-rmse:0.655858                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.384254\ttest-rmse:0.65229                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.365578\ttest-rmse:0.650074                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.43647\ttest-rmse:4.43631                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.80995\ttest-rmse:3.81228                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:3.27609\ttest-rmse:3.27851                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.82134\ttest-rmse:2.82805                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:2.43392\ttest-rmse:2.44226                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:2.10487\ttest-rmse:2.11781                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.82539\ttest-rmse:1.84352                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:1.58824\ttest-rmse:1.6131                                                                                \n",
      "\n",
      "[8]\ttrain-rmse:1.3884\ttest-rmse:1.42065                                                                                \n",
      "\n",
      "[9]\ttrain-rmse:1.21935\ttest-rmse:1.25971                                                                               \n",
      "\n",
      "[0]\ttrain-rmse:3.40196\ttest-rmse:3.4023                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.26583\ttest-rmse:2.27763                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.53958\ttest-rmse:1.57125                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.08198\ttest-rmse:1.1406                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:0.797277\ttest-rmse:0.892794                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.62493\ttest-rmse:0.763616                                                                              \n",
      "\n",
      "[6]\ttrain-rmse:0.516629\ttest-rmse:0.692131                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.455292\ttest-rmse:0.662744                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.412637\ttest-rmse:0.648871                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.386688\ttest-rmse:0.645814                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.74606\ttest-rmse:3.74616                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.73034\ttest-rmse:2.73667                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.00749\ttest-rmse:2.01893                                                                               \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[3]\ttrain-rmse:1.49589\ttest-rmse:1.52911                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.13346\ttest-rmse:1.19323                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.872366\ttest-rmse:0.980147                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.682151\ttest-rmse:0.841906                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.53986\ttest-rmse:0.759743                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.435898\ttest-rmse:0.710544                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.360772\ttest-rmse:0.679507                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.27256\ttest-rmse:3.27302                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.10389\ttest-rmse:2.11701                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.3876\ttest-rmse:1.43847                                                                                \n",
      "\n",
      "[3]\ttrain-rmse:0.950561\ttest-rmse:1.04989                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.682285\ttest-rmse:0.843239                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.516582\ttest-rmse:0.742959                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.411422\ttest-rmse:0.694533                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.345436\ttest-rmse:0.675363                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.300934\ttest-rmse:0.663946                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.269063\ttest-rmse:0.657616                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.37148\ttest-rmse:4.37133                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.69997\ttest-rmse:3.70258                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:3.13673\ttest-rmse:3.13958                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:2.66444\ttest-rmse:2.67214                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:2.26824\ttest-rmse:2.27884                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:1.93689\ttest-rmse:1.95477                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:1.65999\ttest-rmse:1.68617                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:1.42729\ttest-rmse:1.46745                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:1.2321\ttest-rmse:1.28941                                                                                \n",
      "\n",
      "[9]\ttrain-rmse:1.06796\ttest-rmse:1.1475                                                                                \n",
      "\n",
      "[0]\ttrain-rmse:3.84014\ttest-rmse:3.84019                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.86594\ttest-rmse:2.87116                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.15415\ttest-rmse:2.16172                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.63722\ttest-rmse:1.65917                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.2652\ttest-rmse:1.30217                                                                                \n",
      "\n",
      "[5]\ttrain-rmse:0.998222\ttest-rmse:1.0573                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:0.81214\ttest-rmse:0.893189                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.683292\ttest-rmse:0.787539                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.596921\ttest-rmse:0.722657                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.538867\ttest-rmse:0.680011                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.8336\ttest-rmse:2.83462                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.60917\ttest-rmse:1.6319                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:0.982835\ttest-rmse:1.03597                                                                              \n",
      "\n",
      "[3]\ttrain-rmse:0.671921\ttest-rmse:0.786541                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.526828\ttest-rmse:0.696774                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.454804\ttest-rmse:0.673556                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.412692\ttest-rmse:0.663737                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.377604\ttest-rmse:0.660383                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.357904\ttest-rmse:0.657477                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.342432\ttest-rmse:0.657964                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.48063\ttest-rmse:3.48091                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.36773\ttest-rmse:2.37789                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.63708\ttest-rmse:1.66562                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.16077\ttest-rmse:1.22062                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.842796\ttest-rmse:0.961192                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.629284\ttest-rmse:0.817118                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.484516\ttest-rmse:0.736413                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.379358\ttest-rmse:0.689636                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.304318\ttest-rmse:0.668914                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.249404\ttest-rmse:0.657641                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.05798\ttest-rmse:3.05868                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.85172\ttest-rmse:1.86762                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.17486\ttest-rmse:1.21665                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.810141\ttest-rmse:0.886477                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.623595\ttest-rmse:0.742268                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.528515\ttest-rmse:0.685103                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.479534\ttest-rmse:0.659852                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.448101\ttest-rmse:0.655537                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.428866\ttest-rmse:0.653589                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.41314\ttest-rmse:0.650614                                                                              \n",
      "\n",
      "[0]\ttrain-rmse:3.27117\ttest-rmse:3.27164                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.1022\ttest-rmse:2.11533                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.38607\ttest-rmse:1.437                                                                                 \n",
      "\n",
      "[3]\ttrain-rmse:0.949262\ttest-rmse:1.04876                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.682345\ttest-rmse:0.842561                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.516315\ttest-rmse:0.744039                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.411444\ttest-rmse:0.696236                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.34496\ttest-rmse:0.677143                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.30309\ttest-rmse:0.667492                                                                              \n",
      "\n",
      "[9]\ttrain-rmse:0.274072\ttest-rmse:0.661708                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.99047\ttest-rmse:3.99044                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:3.09006\ttest-rmse:3.09353                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:2.40422\ttest-rmse:2.40958                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.88339\ttest-rmse:1.90003                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.4881\ttest-rmse:1.52469                                                                                \n",
      "\n",
      "[5]\ttrain-rmse:1.19092\ttest-rmse:1.25013                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:0.964891\ttest-rmse:1.04896                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.79628\ttest-rmse:0.910544                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.668733\ttest-rmse:0.817385                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.572385\ttest-rmse:0.753602                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.75349\ttest-rmse:3.75358                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.7409\ttest-rmse:2.74718                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:2.01887\ttest-rmse:2.03014                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.50679\ttest-rmse:1.53857                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.14359\ttest-rmse:1.19856                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.884079\ttest-rmse:0.98354                                                                              \n",
      "\n",
      "[6]\ttrain-rmse:0.695865\ttest-rmse:0.847269                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.561334\ttest-rmse:0.759784                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.458363\ttest-rmse:0.709718                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.384026\ttest-rmse:0.680234                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:3.58616\ttest-rmse:3.58636                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.50829\ttest-rmse:2.51762                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.77794\ttest-rmse:1.79791                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.28762\ttest-rmse:1.32838                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:0.960941\ttest-rmse:1.02967                                                                              \n",
      "\n",
      "[5]\ttrain-rmse:0.746739\ttest-rmse:0.849933                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.606939\ttest-rmse:0.750417                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.518377\ttest-rmse:0.694567                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.460833\ttest-rmse:0.663391                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.420494\ttest-rmse:0.649406                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:4.88809\ttest-rmse:4.88788                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:4.62119\ttest-rmse:4.62115                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:4.36939\ttest-rmse:4.36969                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:4.1319\ttest-rmse:4.13206                                                                                \n",
      "\n",
      "[4]\ttrain-rmse:3.90782\ttest-rmse:3.90845                                                                               \n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[5]\ttrain-rmse:3.69652\ttest-rmse:3.69721                                                                               \n",
      "\n",
      "[6]\ttrain-rmse:3.49725\ttest-rmse:3.49822                                                                               \n",
      "\n",
      "[7]\ttrain-rmse:3.30947\ttest-rmse:3.31103                                                                               \n",
      "\n",
      "[8]\ttrain-rmse:3.13221\ttest-rmse:3.13469                                                                               \n",
      "\n",
      "[9]\ttrain-rmse:2.96513\ttest-rmse:2.96872                                                                               \n",
      "\n",
      "[0]\ttrain-rmse:2.7418\ttest-rmse:2.74297                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.51558\ttest-rmse:1.54073                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:0.902409\ttest-rmse:0.989948                                                                             \n",
      "\n",
      "[3]\ttrain-rmse:0.577568\ttest-rmse:0.781451                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.404092\ttest-rmse:0.704658                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.302464\ttest-rmse:0.677497                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.235749\ttest-rmse:0.675533                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.182735\ttest-rmse:0.67591                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.151396\ttest-rmse:0.677583                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.121368\ttest-rmse:0.678538                                                                             \n",
      "\n",
      "Stopping. Best iteration:                                                                                              \n",
      "[6]\ttrain-rmse:0.235749+0.021492\ttest-rmse:0.675533+0.0642438\n",
      "\n",
      "\n",
      "[0]\ttrain-rmse:3.6836\ttest-rmse:3.68374                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:2.64229\ttest-rmse:2.65079                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.91502\ttest-rmse:1.92928                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:1.41013\ttest-rmse:1.44557                                                                               \n",
      "\n",
      "[4]\ttrain-rmse:1.06242\ttest-rmse:1.13009                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.820892\ttest-rmse:0.92814                                                                              \n",
      "\n",
      "[6]\ttrain-rmse:0.65593\ttest-rmse:0.801051                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.539329\ttest-rmse:0.728329                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.461229\ttest-rmse:0.689577                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.409826\ttest-rmse:0.668242                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.9446\ttest-rmse:2.94545                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.72705\ttest-rmse:1.74229                                                                               \n",
      "\n",
      "[2]\ttrain-rmse:1.07625\ttest-rmse:1.11889                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.744944\ttest-rmse:0.832802                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.588144\ttest-rmse:0.714692                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.511532\ttest-rmse:0.669961                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.469728\ttest-rmse:0.654318                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.448341\ttest-rmse:0.646392                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.428885\ttest-rmse:0.640743                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.409285\ttest-rmse:0.638775                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.95431\ttest-rmse:2.95515                                                                               \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.73752\ttest-rmse:1.7526                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.08432\ttest-rmse:1.12652                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.750009\ttest-rmse:0.83715                                                                              \n",
      "\n",
      "[4]\ttrain-rmse:0.5909\ttest-rmse:0.716011                                                                               \n",
      "\n",
      "[5]\ttrain-rmse:0.511644\ttest-rmse:0.670205                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.473793\ttest-rmse:0.657631                                                                             \n",
      "\n",
      "[7]\ttrain-rmse:0.450321\ttest-rmse:0.64765                                                                              \n",
      "\n",
      "[8]\ttrain-rmse:0.428746\ttest-rmse:0.643653                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.410876\ttest-rmse:0.640234                                                                             \n",
      "\n",
      "[0]\ttrain-rmse:2.95196\ttest-rmse:2.9528                                                                                \n",
      "\n",
      "Multiple eval metrics have been passed: 'test-rmse' will be used for early stopping.                                   \n",
      "\n",
      "\n",
      "Will train until test-rmse hasn't improved in 3 rounds.                                                                \n",
      "\n",
      "[1]\ttrain-rmse:1.73498\ttest-rmse:1.7501                                                                                \n",
      "\n",
      "[2]\ttrain-rmse:1.08235\ttest-rmse:1.12466                                                                               \n",
      "\n",
      "[3]\ttrain-rmse:0.748772\ttest-rmse:0.836087                                                                             \n",
      "\n",
      "[4]\ttrain-rmse:0.590231\ttest-rmse:0.715512                                                                             \n",
      "\n",
      "[5]\ttrain-rmse:0.511304\ttest-rmse:0.670014                                                                             \n",
      "\n",
      "[6]\ttrain-rmse:0.473608\ttest-rmse:0.65754                                                                              \n",
      "\n",
      "[7]\ttrain-rmse:0.450189\ttest-rmse:0.647339                                                                             \n",
      "\n",
      "[8]\ttrain-rmse:0.428634\ttest-rmse:0.642954                                                                             \n",
      "\n",
      "[9]\ttrain-rmse:0.410831\ttest-rmse:0.64032                                                                              \n",
      "\n",
      "100%|████████████████████████████████████████████████████████████| 50/50 [00:09<00:00,  6.29it/s, best loss: 0.6387752]\n",
      "\n",
      "展示hyperopt获取的最佳结果，但是要注意的是我们对hyperopt最初的取值范围做过一次转换\n",
      "{'learning_rate': 0.44307009444016143, 'max_depth': 0}\n"
     ]
    }
   ],
   "source": [
    "from numpy.random import RandomState\n",
    "\n",
    "params_space = {\n",
    "    'max_depth': hyperopt.hp.randint('max_depth', 6),\n",
    "    'learning_rate': hyperopt.hp.uniform('learning_rate', 1e-3, 5e-1),\n",
    "}\n",
    "\n",
    "trials = hyperopt.Trials()\n",
    "\n",
    "best = hyperopt.fmin(\n",
    "    hyperopt_objective,\n",
    "    space=params_space,\n",
    "    algo=hyperopt.tpe.suggest,\n",
    "    max_evals=50,\n",
    "    trials=trials,\n",
    "    rstate=RandomState(123)\n",
    ")\n",
    "\n",
    "print(\"\\n展示hyperopt获取的最佳结果，但是要注意的是我们对hyperopt最初的取值范围做过一次转换\")\n",
    "print(best)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用sklearn进行交叉验证\n",
    "\n",
    "这里要注意的是运算速度上也是有一定差距的。而且可能很大。本例运行于我个人的笔记本时候，与sklearn结合的方案约是完全使用xgboost的3-6倍"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:31:45.035278Z",
     "start_time": "2019-04-12T10:31:45.028297Z"
    }
   },
   "outputs": [],
   "source": [
    "def XGBRegressor_CV(params):\n",
    "    from sklearn.model_selection import cross_val_score\n",
    "    \n",
    "    model = xgb.XGBRegressor(\n",
    "        max_depth=int(params['max_depth'])+5,\n",
    "        learning_rate=params['learning_rate'],\n",
    "        silent=1,\n",
    "        objective='reg:linear',\n",
    "        eval_metric='rmse',\n",
    "        seed=619,\n",
    "        nthread=-1,\n",
    "        early_stopping_rounds=3\n",
    "    )\n",
    "\n",
    "#     x_train, x_predict, y_train, y_predict\n",
    "    metric = cross_val_score(model, x_train, y_train, cv=5, scoring=\"neg_mean_squared_error\")\n",
    "\n",
    "    return np.mean(-metric)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-04-12T10:32:20.371763Z",
     "start_time": "2019-04-12T10:31:45.593784Z"
    },
    "code_folding": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "100%|██████████████████████████████████████████████████| 50/50 [00:34<00:00,  1.55it/s, best loss: 0.37836451650019465]\n",
      "\n",
      "展示hyperopt获取的最佳结果，但是要注意的是我们对hyperopt最初的取值范围做过一次转换\n",
      "{'learning_rate': 0.05869477251939005, 'max_depth': 3}\n"
     ]
    }
   ],
   "source": [
    "trials_2 = hyperopt.Trials()\n",
    "\n",
    "best_2 = hyperopt.fmin(\n",
    "    XGBRegressor_CV,\n",
    "    space=params_space,\n",
    "    algo=hyperopt.tpe.suggest,\n",
    "    max_evals=50,\n",
    "    trials=trials_2,\n",
    "    rstate=RandomState(123)\n",
    ")\n",
    "\n",
    "print(\"\\n展示hyperopt获取的最佳结果，但是要注意的是我们对hyperopt最初的取值范围做过一次转换\")\n",
    "print(best_2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.0"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
